✨ Concurrent E2E Test Execution#2675
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
This PR updates the E2E Godog test runner to take advantage of scenario isolation (from #2651) by running most scenarios in parallel while reserving tagged scenarios for a serial pass, and then printing a combined failure summary.
Changes:
- Split E2E execution into two Godog runs: parallel (excluding
@Serial) followed by serial (@Serialonly), with aggregated failure output. - Tag TLS E2E feature(s) with
@Serialso they run only in the serial phase. - Reduce
test-experimental-e2etimeout from 20m to 10m.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/e2e/features_test.go | Runs E2E suite in parallel + serial phases and prints an aggregated failure summary. |
| test/e2e/features/tls.feature | Marks TLS feature as @Serial to exclude it from the parallel phase. |
| Makefile | Adjusts experimental E2E timeout setting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2675 +/- ##
==========================================
+ Coverage 67.98% 68.00% +0.01%
==========================================
Files 144 144
Lines 10595 10595
==========================================
+ Hits 7203 7205 +2
+ Misses 2870 2869 -1
+ Partials 522 521 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Create buffers to capture output for final summary | ||
| var parallelBuf, serialBuf bytes.Buffer | ||
|
|
||
| parallelOpts := cliOpts | ||
| if parallelOpts.Concurrency == 1 { | ||
| // Override default concurrency value with 100; otherwise use whatever was provided by CLI | ||
| parallelOpts.Concurrency = 100 | ||
| } | ||
| parallelOpts.Tags = "~@Serial" | ||
| // Write to both specified output (live to stdout, by default) and buffer (for summary) | ||
| parallelOpts.Output = io.MultiWriter(parallelOpts.Output, ¶llelBuf) | ||
| // run tests concurrently |
There was a problem hiding this comment.
This duplicates the entire godog console output into in-memory buffers (parallelBuf/serialBuf) via io.MultiWriter. With many scenarios (and especially at high concurrency), this can consume a lot of memory and slow down CI due to extra copying. Consider limiting what’s captured (e.g., only buffering after the "--- Failed steps:" marker, keeping only the last N lines, or writing captured output to a temp file and only reading it back on failure).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| go test test/e2e/features_test.go --godog.tags="@WebhookProviderCertManager" | ||
| ``` | ||
|
|
||
| Note that setting the tags in this way will disable the automatic test parallelization. If running in parallel with custom tags is desired, set `--godog.concurrency=100` for instance to re-enable. If this is done adding `&& ~@Serial` to the tags as well is highly recommended: |
There was a problem hiding this comment.
The sentence starting with “If this is done…” is grammatically awkward and a bit hard to parse. Consider rephrasing (e.g., “If you do this, adding && ~@Serial to the tag expression is highly recommended…”) to make the guidance clearer.
| Note that setting the tags in this way will disable the automatic test parallelization. If running in parallel with custom tags is desired, set `--godog.concurrency=100` for instance to re-enable. If this is done adding `&& ~@Serial` to the tags as well is highly recommended: | |
| Note that setting the tags in this way will disable the automatic test parallelization. If running in parallel with custom tags is desired, set `--godog.concurrency=100` for instance to re-enable. If you do this, adding `&& ~@Serial` to the tags is also highly recommended: |
|
There are also tests in |
Good callout; I haven't seen that fail yet, so it might be fine to leave it, or we could just get ahead of it. I'm fine either way. |
|
Let's get ahead of it, since the proxy tests may be disruptive. |
Takes advantage of changes made to isolate test runs to execute as many tests in parallel as possible. For tests that must be run serially, the @serial tag has been added to the beginning of relevant feature file(s). Signed-off-by: Daniel Franz <dfranz@redhat.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Done! |
|
/approve |
openshift-ci
Bot
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rashmigottipati, tmshort The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Description
Takes advantage of changes made to isolate test runs (#2651) to execute as many tests in parallel as possible. For tests that must be run serially, the
@Serialtag has been added to the beginning of relevant feature file(s).Tests are now executed in two steps: parallel followed by serial. Since this creates two distinct test outputs, the final output has been modified to combine any step failures into one smaller output at the end, while still maintaining live output to stdout. Ideally this could be formatted into a nicer looking markdown output in the test summary that we already output, but I'm saving that for a future PR.
Reviewer Checklist